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Abstract -After nearly a decade of experience, we reflect on the 
principles and lessons which have emerged in the field of packet com¬ 
munications. We begin by identifying the need for efficient resource 
sharing and review the original and recurring difficulties we had in 
achieving this goal in packet networks. We then discuss various lessons 
learned in the areas of: deadlocks; degradations; distributed control; 
broadcast channels; and hierarchical design. The principles which we 
discuss have to do with: the efficiency of large system; the switching 
computer; network constraints; distributed control; flow control; stale' 
protocols; and designers not yet experienced in packet communications. 
Throughout the paper, we identify various open issues which remain to 
be solved in packet communications. 

I. Introduction 

HAT IS IT WE now know about packet communica¬ 
tions that we did not know a decade ago? What made 
the problem difficult, and why were the solutions not 
immediately apparent to us in the late 1960’s? Whereas the 
answers to these questions may entice the system designer (in¬ 
deed, I, for one, delight in such investigations), why should 
the network user care a whit? To most users (and, alas, to 
many designers), communications is simply a nuisance and they 
would just as soon ignore those problems and get on with the 
“real” issues of information processing! 

In this paper (and in conjunction with the other papers in 
this Special Issue of the Proceedings), we hope to answer 
some of these questions. We will identify the need for resource 
sharing, explain why the problem of efficient resource sharing 
is hard, and why it must be understood, review some of the 
lessons we learned (mostly from the three ARPA packet net¬ 
works), and then, finally, state some principles which have 
evolved from the study and extensive use of packet communica¬ 
tions. 

II. Resource Sharing 

A privately owned automobile is usually a waste of money! 
Perhaps 90 percent of the time itis;idly parked and'not in use. 
However, its “convenience” is.so seductive that few can resist 
the temptation to own one. Whenfthe price of such a poorly 
utilized device is astronomically high, we do refuse the temp¬ 
tation (how many of us own private jet aircraft?). On the 
other hand, when the cost is extremely low, we are obliged to 
own such resources (we all own idle pencils). 

An information processing system consists of many poorly 
utilized resources. (A resource is simply a device which can 
perform work for us at a finite rate.) For example, in an in¬ 
formation processing system, there is the CPU, the main mem¬ 
ory, the disk, the data channels, the terminals, the printer, etc. 
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One of the major system advances of the early 1960’s 
development of multiaccess time-sharing systems in 
computer system resources were made available to a Ll- 
population of users, each of whom had relatively small d crnUl!t 
(i.e., the ratio of their peak demands to their average dcmu,& 
was very high) but who collectively presented a total ilcrrj.i,i 
profile which was relatively smooth and of medium to 
utilization. This was an example of the advantages in i. 
gained through the smoothing effect of a large popuU:m 
(i.e., the “law of large numbers”) [1], The need forreso-j-., 
sharing is present in many many systems (e.g., the sharco u» 
of public jet aircraft among a large population of users). 

In computer communication systems we have a great 
for sharing expensive resources among a collection of 
peak-to-average (i.e., “bursty”) users [1], In Fig.l wedn, -la¬ 
the structure of a computer network in which we can ulr.-T- 
three kinds of resources: 

1) the terminals directly available to the user and the <•>■■■■ 
munications resources required to connect those terniirua 
their HOST computers or directly into the network (via i:n 
in the ARPANET, for example—this is an expensive poriios 
the system and it is generally difficult to employ c.xic.-.v"’ 
resource sharing here due to the relative sparseness of the im 
sources; 

2) the HOST machines themselves which provide ihc jt 
formation processing services—here multiaccess time sharai 
provides the mechanism for efficient resource sharing: 

3) the communications subnetwork, consisting of 
munication trunks and software switches, whose function t » 
to provide the data communication service for the exclnr-r 
data and control among the other devices. 

The HOST machines in 2) above contain hardware anJ 
ware resources (in the form of application programs anJ 
files) whose sharing comes under the topic of time sharint. * 
dwell no further on these'resources. Rather, we shall 
attention on those portions of the computer communiui *-' 1 
system where packet communications has had an impr¬ 
int pact. Perhaps the most visible component is that of the 
munications subnetwork listed in item 3) above. Here r** ^ 
communications first demonstrated its enormousefficu’ n ^ ^ 
the form of the ARPANET in the early 1970’s(the d^a-^ 
computer networks). The communication resources 1 ^ 
shared in this case are storage capacity at the nodal s "'“ 
(these switches are called IMP’S in the ARPANET), P r0i( ^ 
capacity in the nodal switches, and communications capt (U 
the trunks connecting these switches. Packet switching 
environment has proven to be a major technological 
through in providing cost effective data communications^ ^ 
information processing systems. Deep in the backbone ^ 
packet-switched networks there is a need for long-n ^ ^ 
capacity inexpensive communications, and it is here w 
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'We quickly found that many of our old techniques could not 
be directly applied to packet network design and that new 
techniques had to be developed; these new techniques turned 
out to be of great generality and have led to principles and to 
understanding which are sure to benefit us for many years to 
come. One of our first tasks was to develop tools which would 
allow us to analyze the performance of a given network. This 
involved evaluating the delay-throughput profile for networks. 
Basically, this is a queueing problem in a network environment. 
It had earlier been recognized [8] that the probabilistic com¬ 
plexities one encounters in computer networks are extremely 
difficult. One of the simplest analytical questions involving 
the solution of two nodes in tandem was first posed at that 
time in 1964 and has only been satisfactorily answered (in the 
queueing theoretic sense) within the past year [9]; this, note, 
is for the simplest problem. Indeed we have come to realize 
that an exact solution for the delay-throughput profile is prob¬ 
ably hopeless in a complex network environment. Fortunately 
suitable approximations [1], [8] have been developed which 
permit one to predict the performance of given networks with 
a high degree of accuracy. More than that, these approximate 
tools allow us to expose and understand the phenomenological 
behavior of networks. 

The astute reader will observe that the resource sharing prob¬ 
lem stated above sounds very much like the problem faced in 
the design of time sharing systems. Surely, with time sharing, 
we are faced with the problem of sharing resources among 
asynchronous processes which behave in a bursty fashion. The 
major difference between the two problems, however, is that 
our problem exists in a geographically distributed environment 
which requires expensive communication resources in the com¬ 
munications and coordination functions. The implications 
here are strong. For example, when communication is cheap, 
then wide-band communications can be obtained with ex¬ 
tremely small delays; such is the case, for example, within the 
resources of a local operating system connected together by a 
data bus. In a regional or nationwide network subject to the 
relatively expensive cost of telecommunications, we find that 
typical bandwidths are many orders of magnitude less than that 
in a local time-sharing environment, and the delays are many 
orders of magnitude greater. Furthermore, the control of these 
processes in the time sharing environment can be very tightly 
coupled if desired or left loosely coupled if there is sufficient 
reason; in the network environment we typically find our pro¬ 
cesses are very loosely coupled due-to the difficulty of tighten¬ 
ing the control between thenr(indeed, the inherent delay due 
to the finite speed of light is a .fundamental limitation on the 
tight coupling of remote processes). The overhead in the time 
sharing system is variable and may be very high with poor system 
design (for example, thrashing) but may be made small with 
clever design. In the network environment, for a variety of 
reasons, we find that the overhead due to packet headers, 
control information and resource allocation tends to be 
relatively high. These comparisons are summarized in Table 1. 

Not only do we have extremes in communications cost 
between these two systems, we also have a significant difference 
in the reliability of that communications. Indeed, in the local 
time shared system, the process-to-process communication is 
usually assumed to be reliable and therefore the acknowledg¬ 
ment procedure (if any exists) is simple and tends to be invoked 
only under exceptional circumstances. On the other hand, in 
the distributed computer network environment, communica¬ 
tions reliability is not assumed, and, therefore, an elaborate 



TABLE: 

Asynchronous Process-to-Process Communication and Cos 
Cost Comparison Between Local Processes in a Timt-Sh, 
System and Distributed Processes in a Network 



Multiacess 

Time-Shared 

Systems 

Geographical}, 
Distributed 
Computer Nct» wll 

Typical bandwidth 

megabytes/sec 

kilobits/sec 

Typical communica- 

fractions of a micro- 

tens to iiur,drc,!i 

tions delay 

second 

millisecond* 

Process-process 



coupling 

tight to loose 

typically loose 

Overhead due to 

variable (typically 

variable flypicaih 

system control 

low) 

high) 

acknowledgment procedure (often including timeouts and mLir 

defaults) is usually 

invoked. We are here reminded o( u,. 


“two-army” problem. This is the problem where two Ku 
armies are each poised on opposite hills preparing to aiuu , 
single red army in the valley. The red army can conquer cun.- 
of the blue armies separately but will fall to defeat if both 
armies attack simultaneously. The blue armies community, 
with each other over an unreliable communications svsinr 
The problem is to coordinate the two blue armies so that tbr. 
will attack simultaneously. Let us assume that Blue Am:. 
(Bl) sends a message to Blue Army 2 (B2) indicating that iv- 
should jointly attack at noon the next day. Clearly Bl r .x 
await a positive acknowledgment from B2 that the conmu.-M 
was properly received; were Bl to attack without such n 
acknowledgment, then the possibility exists that B2 did > 
receive the message correctly, in which case Bl is subjo.: ; 
the certain annihilation of his army. If B2 properly recci’” 
the command and acknowledges it, then he must awaii >■ 
acknowledgment of the acknowledgment from Bl, for :f 9 
did not receive his acknowledgment then we know Bl will 
attack and B2’s attack will then be doomed tq failure. D-* 
argument continues in a circular fashion where acks ofi-P 
of acks ... are continually transmitted with no action c«- 
being taken; the two blue armies can never get perfee: ■* 
synchronized with certainty using this unreliable communa-* 
tions system. 

We see therefore that the new culprit in resource sharing 1 
distributed environment is the fact that the resources j- -1 
distributed and cannot easily be assigned to the demands ,1- 
deed,, in previous resource allocation problems, winch o.-- 
come under the name of scheduling algorithms, we made 5 ■* 
assumption, namely, that the scheduling information coul- 
passed around for free. That is, the competing demands 
organize themselves into a cooperating queue at no price 

fortunately, in the distributed environment, the cost of orp»—-^ 

ing demands into a cooperating queue may be very l ar £ c .. 
in one fashion or another nature will make you pay a price 


The problem of resource sharing in a distributed eni> ronn: ^ 
manifests itself in the routing control and flow control r r ^ 
lems. The problem of flow control is to regulate the r31 ^ 
which data crosses the boundary of the communication 5 ^ 
network (both into and out of the network). The prob 
routing control is to efficiently transport the data ( w 1 ^ 
been admitted by the flow control procedure) across 
work to its destination. In 1967 we were aware oi m ^ 
tication needed in the routing procedure, but were re ^ 
ignorant of the need for effective flow control (see , 
These two problems are difficult because we are dealing 
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irol procedure in a distributed environment subject to 
* oin delays in passing that control information around 
”ord er to contr °l the random demands. The purpose of both 
1 -edures is to efficiently use the network resources (IMP 
f -r 3 ge. processing capacity, and communications ca- 

r '.|iy) In achieving this goal one must attempt to control 
aiestion, route data around busy or defective portions of 
■ network, and in general must adaptively assign capacity to 
" j a ta traffic flow in an efficient, dynamic way. These are 
.j control problems and represent a class which has not 
^n adequately studied up to and including the present time. 
,. have come to learn that distributed control is a sophisticated 
■blem. Below we return to the issues involved in distributed 
,urce allocation and sharing. For the moment let us in- 
•juce some of the other sources of complexity in packet 
■un unicat ions. 

la any distributed communications system design one is faced 
•h a topological design problem. The problem basically is, 
,-n.a set of constraints to meet, find that topological design 
•jaure which meets these constraints at least cost. The field 
network flow theory addresses itself to such problems and 
: salient feature of this theory is that most of its problems 
.-.not be solved! To exhaustively search over all possible 
pological designs for a given problem is certainly not a solu- 
, n since the number of possible alternatives to consider can 
auly exceed the number of atoms in the universe even for 
inively small problems. (For example, if at some stage you 
:.>i consider all permutations of 20 objects, then a computer 
i.jld take more than 75 000 years to process all 20! cases even 
. :i could examine one million cases per second.) Rather, a 
• ijlion consists of elegant search procedures which are com- 
-.uiioually efficient and which find the optimal topology 
: the given problem. Very few problems in network flow 
t:ory yield to such efficient algorithms. Rather, one gets 
: end the combinatorial complexity naturally inherent in 
vie problems by accepting suboptimal solutions. (Beware! 
i uiboptimal solution to a problem is simply the result of a 
"xedure which examines a subset of possible solutions and 
vts the best of those examined-this suboptimal solution 
-iy or may not be close to the optimal.) The trick here is to 
efficient heuristic search procedures which come close to 
V optimal rapidly. Over the past decade, efficient procedures 
u, t been developed in many cases and new procedures are 
■etstantly being investigated for the topological design problem. 
Mother source of difficulty in the resource sharing problem 
* ;n defining the appropriate performance measure. For ex- 
c ?le, what is the capaeity-crf a network? It is well-known in 
“^■‘ork flow theory that orte Can easily calculate the capacity 
■•’.the throughput) between- any two pairs of points. What 
1 n ot straightforward is to evaluate the total data-carrying 
a ?icity of a network where throughput is measured in terms 
’f Plages successfully received at their destination. The dif- 
comes about because the capacity of the network 
'■‘ongly depends upon the traffic matrix one assumes for the 
61,1 flowing through that network. For example, if the traffic 
^rix were such that traffic passes only between immediate 
^*f>bors in the topological structure and in an amount equal 
capacity of the line connecting those two, then the net- 
JT t capacity would approach a value equal to the sum of all 
Pnel capacities in the network. This is clearly an upper 
J11 ° to the capacity for any other traffic matrix. Since in 
"' tr 3l we do not know the traffic matrix for a network to be 
■^Sned for future use, how is one to evaluate that capacity? 


Yet another source of difficulty in the network problem is 
that of interfacing terminals and HOSTS to networks as well as 
one network to another network. For example, there, is the 
general issue of virtual circuit service as opposed to datagram 
service. A virtual circuit service presents to the user a com¬ 
munications environment in which all functions appear as if 
she were directly connected between the two points com¬ 
municating (i.e., an orderly and controlled flow is maintained 
whereby data is guaranteed to be correct upon delivery to the 
destination, comes out of the network in the same order in 
which it came in, and a flow control may be applied to that 
transmission to maintain efficient use of network resources 
and of terminal-HOST resources). A datagram service ensures 
none of these things and simply sends blocks of data (packets) 
across the network, not guaranteeing correct delivery (or 
delivery at all), not guaranteeing orderly flow in terms of se¬ 
quencing (packets may arrive at the destination out of order), 
and not enforcing any flow control procedure at the process- 
to-process level. Which of those two services is desirable has 
become an issue of international proportions discussed else¬ 
where in these proceedings [7], [11]. Futhermore, the inter¬ 
connection of two networks presents an enormous and rich 
variety of difficult problems. For example, should one apply 
flow control at the boundary of each network in a tandem 
chain of networks or should one apply flow control from the 
source HOST to the destination HOST across many networks 
simultaneously? If we consider the interconnection of networks 
with different packet sizes, we have the general problem of 
fragmentation whereby long packets get fragmented into smaller 
packets when crossing network boundaries. A variety of other 
very important issues arise in internetting (see [27] for a more 
detailed discussion of internetting). 

Many of the problems we have just presented come about 
due to the distributed nature of our message sources and system 
resources. The problems created by distributed sources are 
very clearly seen in the environment of geographically dispersed 
message sources which communicate with each other over a 
common broadcast channel; in this case, access to the capacity 
of the channel must be carefully coordinated. 

Thus in answering the question, “Why is the problem hard?” 
we have found the following sources of difficulty: 

1) the analytic problems from queueing theory and the 
probabilistic complexity resulting therefrom; 

2) the topological design problems from network flow theory 
and the combinatorial complexity resulting therefrom; 

3) the price for coordinating and sharing resources in a 
geographically distributed environment (the new culprit) 
leading to problems of resource allocation, routing con¬ 
trol, flow, control,, and general process-to-process com¬ 
munication problems. 

IV. Lessons Learned 

After a decade of experience with packet communications it 
is fair that we ask what lessons have we learned and what have 
we come to know about the needs of the user and the questions 
he would like to have answered. So far as the user is concerned 
we shall see as we step through the lessons learned below that 
he cannot insulate himself completely from the underlying 
technology of packet communications. Indeed the service he 
sees is quite different from that which he has with leased lines 
as mentioned above. Moreover, certain decisions will either be 
thrust upon him or accepted by him due to the nature of the 
service offered; if he is unaware of the consequence of setting 
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parameters in that decision-making process then he may 
seriously degrade the performance of the network due to his 
ignorance. Let us now list some of the lessons we learned and 
return to the principles in the following section. 

A, Deadlocks 

In [ 1 ], [12], and [13] we described in detail some of the 
deadlocks and degradations of which we have become aware. 
In this section we simply enumerate and sketch the details of 
the deadlocks. Simply stated, a deadlock (also commonly 
referred to as a lockup) is the unpleasant situation in which 
two (or more) competing demands have each been assigned a 
subset of their necessary resources; neither can proceed until 
one of them collects some additional resources which currently 
are assigned to the other and' neither demand is willing to 
release any resource currently assigned'to him. Deadlocks are 
one of the most serious system malfunctions possible, and one 
must take great care to avoid them or find ways to recover from 
them. It is ironic that flow control procedures by their very 
nature present constraints on the flow of data (e.g., the re¬ 
quirement for proper sequencing), and if the situation ever 
arises whereby the constraint cannot be met, then, by defini¬ 
tion, the flow will stop, and we will have a deadlock! This is 
the philosophical reason why flow control procedures have a 
natural tendency to introduce deadlocks. In this section we 
briefly discuss four ARPANET deadlocks which have come 
to be known as: reassembly lockup; store-and-forward dead¬ 
lock; Christmas lockup ; and piggyback lockup. 

Reassembly lockup, the best known of the ARPANET dead¬ 
lock conditions (and one which was known to exist in the very 
early days of the ARPANET implementation), was due to a 
logical flaw in the original flow-control procedure. In the 
ARPANET, a string of bits to be passed through the network 
is broken into “messages” which are at most approximately 
8000 bits in length. These messages are themselves broken 
into packets which arc at most approximately 1000 bits in 
length. A message requiring more than one packet (up to a 
maximum of eight) is termed a multipacket message and each 
of these packets traverses the network independently; upon 
receipt at the destination node, these packets are “reassembled” 
into their original order and the message itself is recomposed, 
at which time it is ready for delivery out of the network. (A 
more complete description of the-ARPANET protocols may 
be found in [I], [13].) Reassembly lockup occurred when 
partially reassembled messages could not be completely reas¬ 
sembled since the network through which the remaining packets 
had to traverse was congested, thus preventing these packets 
from reaching the destination; that is, each of the destination’s 
neighbors had given all of their relay (store-and-forward) buf¬ 
fers to additional packets (from messages other than those 
being reassembled) heading for that same destination and for 
which there were no unassigned reassembly buffers available. 
Thus the destination was surrounded by a barrier of blocked 
IMP’S which themselves could provide no store-and-forward 
buffers for the needed outstanding packets, and which at the 
same time were prevented from releasing any of their store- 
and-forward buffers since the destination itself refused to ac¬ 
cept these packets due to a lack of reassembly buffers at the 
destination. The deadlock was simply that the remaining 
packets could not reach the destination and complete the 
reassembly until some store-and-forward buffers became free, 
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and the store^and-forward buffers could not be released un[ t | 
remaining packets reached the destination. 

Store-and-forward deadlock is another example of a i 0( ^ 
that can occur in a packet-switched network if no 
precautions arc taken [ 1 ] , [13], The case of “direct" stlj r . 
and-forward lockup is simply a “stand-off.” Let us 
that all store-and-forward buffers in some 1MPM are fil] Cl j 
packets headed toward some destination IMP C throujf, 
neighboring IMP B and that all store-and-forward bufR ; , a 
IMP B are filled with packets headed toward some destinaiu- n 
IMP D through IMP A. Since there is no store-and-for*,., 
buffer space available in either IMP A or B , no packet car 
successfully transmitted between these two IMP’s and a dca,' 
lock situation results. One way to prevent the deadlock n: ; 
prohibit these packets in IMP A from occupying all ih.™ 
stoje-and-forward buffers which are needed by the packcri 
coming in from IMP B (and vice versa) by the introdun:. ,i 
of “buffer classes” as in [14], This is accomplished H 
partitioning the buffers in a switch into classes, say 

■ ■ ■ , Bic, where k is the longest path length in the nctwixt 
A packet arriving at a switch from outside the net has 
only to class B 0 buffers. When a packet arrives at a svnt.j 
after having made k hops so far, it has access to class A, 

■ • • , B/c buffers, etc. Thus, the closer a packet gets to its fir-u 
destination, the more access it has, and therefore the speed:--- 
its passage through the network. It can be proven [14] thj: 
this “buffer class” allocation will prevent direct store-j.-.d 
forward lockup. “Indirect” store-and-forward lockup -j' 
occur when all store-and-forward buffers in a loop of IMP"> 
become filled with packets all of which travel in the ur.* 
direction (clockwise or counter-clockwise) and none of whies 
are within one hop of their destination. Both storc-aml-forsuH 
lockup conditions are far less likely if, as in the ARl’AM T 
more than one path exists between all pairs of communicate 
IMP’S. 

In December 1973, the dormant Christmas lockup comhuc* 
was brought to life. This lockup was exposed by collect. -a 
measurement messages at UCLA from all IMP’S simultaneous 
The Christmas lockup occurred when these measurement n'- 1 
sages arrived at the UCLA IMP for which reassembly nmir- 
had been allocated but for which no reassembly blocks !■-*• 
been given. (A reassembly block is a piece of storage ustu - 
the actual process of reassembling packets back to inessafc* 
These messages had no way to locate their allocated but*'-’ 
since the pointer to an allocated buffer is part of the reassert)- ^ 
block; as a consequence, allocated buffers could never be 
and could never be freed. The difficulty was caused b\ ■ 
system first allocating buffers before it was assured that a <' 1 
sembly block was available. To avoid this kind of lockup, 
sembly blocks are now allocated along with the reassert 1 
buffers for each multipacket message in the ARPANET. 

Piggyback lockup is a deadlock condition which was idenu ^ 
by examining the flow control code and has, as far as we 
never occurred. This lockup condition comes about due <■ 
unfortunate combination of intuitively reasonable go 
plemented in the flow-control procedure. One of these g ^ 
which we have already mentioned, is to deliver message*^ 
destination in the same order that the source received ^ 
The other innocent condition has to do with the reserva ^ 
reassembly storage space at the destination. In order to 
this reservation procedure efficient, it is reasonable ^ at ^ 
the first multipacket message of a long file transfer be r 
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[he reservation. The ARPANET flow control proce- 
|i i^en maintain that reservation for a given file transfer 
, aS successive multipacket messages from that file are 
^tly received in succession at the source IMP. We have 
j t he groundwork for piggyback lockup. Assume that 
. a maximum of eight reassembly buffers in each IMP; 
*-iice of eight is for simplicity, but the argument works 
’ va lue. Let IMP A continually transmit eight-packet 
T (from some long file) to some destination IMP B 
hat all eight reassembly buffers in IMP B are used up by 
insmission of multipacket messages. If now, in the stream 
■•[-packet messages, IMP A sends a single-packet message 
rt 0 f the file transfer) to destination IMP B it will gen- 
n0 t be accepted since there is no reassembly buffer space 
^ The single packet message will therefore be treated 
•■•quest for- buffer allocation (these requests are the 
nism by which reservations are made). This request will 
•< serviced before the RFNM (an end-to-end acknowledg- 
irom the destination to source) for the previous multi- 
;t message has been sent. When the RFNM is generated, 
^irr, all the free reassembly buffers will immediately be 
j-.al to the next multipacket message in the file transfer 
■urient transmission as mentioned above; this allocation 
to be “piggybacked” on the RFNM. In this case, the 
packet message from IMP A that arrives later at IMP B 
! »hich is stored in the eight buffers) cannot be delivered 
0 destination HOST because it is out of order. The single- 
»r.-; message that should be delivered next, however, will 
teach the destination IMP since there is no reassembly 
available, and, therefore, its requested reservation can 
he serviced. Deadlock! A minor modification removes 
tgyback lockup. 

Vrr various deadlock conditions are usually quite easy to 
once they are detected and understood. The trick, 

■ rr, is to expurgate all deadlocks from the control 
•mism ahead of time, either by careful programming 
‘.cult task) or by some automatic checking procedure 
- may be as difficult as proving the correctness of pro- 
• Those deadlocks found in the ARPANET have, to 
*'■ of our knowledge, been eliminated. 

■t(radations 

1 ^gradation is just that, namely, a reduction in the net- 
i level of performance. (Deadlocks are, of course, an ex- 
(orm of degradation which is why we discussed them in 
«P»fate section above.) For our purposes, we shall measure- 
mance in terms of delay and throughput. In this section 
beus s four sources-af—ARPANET degradation, namely: 
*{ in the routing procedure; gaps in transmission; single- 
**' turbulence-, and phasing. 

"Ping comes about due to independent routing decisions 
by separate nodes which cause traffic to return to a 
us >y visited node (or, in a more general definition, causes 
*■ 10 make unnecessarily long excursions on the way to its 
■^hon). of course any reasonable adaptive routing proce- 
4| U detect these loops (through the build-up of queues 
icU V s Perhaps) ar.d will then break the loop and guide 
J ffic directly on to its destination. However, the occur- 
°f loops does cause occasional large delays in the traffic 
,n d in some applications this is quite unacceptable. It is 


^luce the 


a remedy which was introduced in the ARPANET 


occurrence of loops, in fact made them worse in 


the sense that whereas they occurred less frequently, when 
they did occur, they persisted for a longer time. Some loop-free 
algorithms have recently been published [15], [16], 

The next degradation we wish to discuss is the occurrence 
of gaps in the message flow. These gaps come about due to a 
limitation on the number of messages in transit which the 
network will allow. Assume that between any source and 
destination, the network will permit n messages in flight at a 
time. If n messages are in flight, then the next one may not 
proceed until an end-to-end acknowledgment is returned back 
at the source for any one of the n outstanding messages. We 
now observe that if the round-trip delay (i.e., the time required 
to send a message across the network in the forward direction 
and to return its acknowledgment in the reverse direction) is 
greater than the time it takes to feed the n messages into the 
network, then the source will be blocked awaiting ack’s to 
release further messages. This clearly will introduce gaps in 
the message flow resulting in a reduced throughput which we 
might classify as a mild form of degradation. 

We now come to the issue of single packet turbulence as 
observed in the ARPANET. We note that “regular” single¬ 
packet messages in the early ARPANET were not accepted by 
the destination if they arrived out of order. Rather, they were 
then treated as a request for the allocation of one reassembly 
buffer. Therefore if, in a stream of single-packet messages, 
packet p arrived out-of-order (say it arrived after packetp + 3), 
then packets p + 1, p + 2, and p + 3 would all be discarded at 
the destination, and only after packet p arrived would a single 
packet buffer be allocated to message p + 1. This allocation 
piggybacked on the end-to-end ack for packet p, and when it 
arrived at the source IMP, it then caused a retransmission of 
the discarded packet p + 1 (which had been stored in the 
source).' Of course any packets arrving at the destination after 
packet /* + 3, but before p + 1 arrived in order, would them¬ 
selves be discarded. When packet p + 1 finally arrived for the 
second time at the destination IMP it was then in order and this 
caused an allocation of a single-packet buffer to packet/? + 2, 
etc. The net result was that only one packet would be deliver¬ 
able to the destination per round-trip time along this path; 
had no packets been received out-of-order, then we would have 
been pumping at a rate close to n packets per round trip time 
(if the maximum number in transit n could fit into the pipe). 
Observe that once a single packet arrived out-of-order in this 
stream, then the degradation from n to 1 packets per round- 
trip time would persist forever until either some supervisory 
action was taken or until-the traffic stream ceased and began 
again from a fresh start in the future. We refer to this effect 
as “single-packet turbulence,” and it was observed in the 
ARPANET.as described in [17], The need to handle a con¬ 
tinuous stream' of traffic (e.g., packetized speech) was 
recognized some time ago and resulted in the definition of 
“irregular” packets known as type 3 packets (as contrasted 
to “regular” type 0 packets); these packets are allowed to 
be delivered out of order, receive no end-to-end acknowledg¬ 
ment, and are generally handled in a much more relaxed 
fashion. 

The last degradation we discuss is known as “phasing.” In a 
typical packet network, more than one resource is often re¬ 
quired before a message is allowed to flow across that net¬ 
work. For example, some required resources may be: a message 
number; storage space at the source; storage space at the 
destination, etc. Tokens move around the network passing out 
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these resources in some distributed fashion. Phasing is the 
phenomenon whereby enough free tokens are available in the 
network to permit message flow, but, the proper mix of tokens 
is not available simultaneously at the proper location in the 
net. The delay in gathering these tokens represents a degrada¬ 
tion [1],[18], 

Fortunately, the degradations here described have been 
remedied in the ARPANET and in later networks. 

C. Lessons of Distributed Control 

We have had “lessons” in two areas of distributed control. 
The first has to do with flow control, and it is simply the 
observation that flow control procedures are rather difficult 
to invent and extremely difficult to analyze. The deadlocks 
and degradations referred to the in previous subsections were 
principally due to flow control failures (and occasionally rout¬ 
ing control failures). To data there is no satisfactory theory or 
procedure for designing efficient flow control procedures, much 
less evaluating their performance, proving they contain no 
deadlocks, and proving that they are correct. Attempts in this 
direction are currently under way. 

An important lesson we have learned with flow control is 
that a packet communications system offers an opportunity 
for passing data between two devices of (very) different speeds. 
We can effectively connect a slow-speed teletype to an enor¬ 
mously high-speed memory channel over a packet network and 
apply flow control procedures which protect the two devices 
from each other as well as protecting the net from both. 
Specifically, we must not drown the teletype with a flood of 
high-speed input, nor must we “nickel-and-dime” a high per¬ 
formance HOST to death with incessant interrupts, nor must 
we use the network as a storage medium for megabytes of data. 
Flow control mechanisms provide the means to accomplish 
this; the trick is to do it well. 

The second area of distributed control in packet communica¬ 
tions has to do with the routing control. The ARPANET, and 
many of the networks which have since based their design on 
the packet-switching technology which emerged from the 
ARPANET experiment, employ an adaptive routing procedure 
with distributed control. In such a procedure, routes for the 
data traffic are not preassigned but rather are dynamically as¬ 
signed when they are needed according to the current network 
status. Control packets (called routing update packets) which 
describe the state of the network to some degree are passed 
back and forth between neighboring IMP’S in some fashion and' 
current queue lengths and congestion measures are added to 
these updates by each IMP. The' ARPANET employs a periodic 
update routing procedure whose rate depends upon channel 
utilization and line speed. The updates passing between IMP’s 
have no priority in competing for the processing capacity of 
the CPU at the IMP’S but do have priority in the queue discip¬ 
line feeding the output modems between IMP’s. An important 
lesson learned is that giving low priority to the processing of 
routing updates appears to be advantageous since the processing 
load on the CPU is rather large and prevents the further dis¬ 
patching of arriving packets to output queues [19]. Another 
routing lesson we have learned is that frequent updates cause 
background congestion in a network which may be intolerably 
high even in the absence of other data traffic; the update proce¬ 
dure and update rate must be carefully chosen. A number of 
alternatives to periodic updates have been suggested [ 1 ] in¬ 
cluding such things as aperiodic updating (send updates only 
when status information has crossed certain thresholds and 
then send it immediately); and purely local information for 


-- 

routing decisions based on queue lengths within a gj v 
and knowledge of the current topology. Fuitherrno/' 
care is taken, there is a tendency for looping to occur 
distributed control algorithms; looping can be prevcm'V 
more sophisticated algorithms [15]. ^ 

One of the lessons which is now beginning to emerge «- 
the most important advantage of distributed control i‘ j 
routing is its ability to automatically sense confi r ..[., 
changes in the network; these configuration changes - 
planned or accidental as for example the result of 3 Lj, 
IMP failure. This is important for two reasons: first 
configuration changes do happen often enough so iiu- 
requirement for a centralized control evaluatius no* . r , L . 
tables based on the current configuration would be 3 - 
mously complex task from an administrative point o! 
second because it is specifically at times of configu.- 1 'u, 
changes when drastic network action must be taken anc 
then is the adaptive routing procedure really called u,-., 
do serious work (it is not yet clear to what extent the rou:, 
algorithm should adapt to statistical fluctuations in trai.':. 

Without diminishing the result of these lessons, it « it, 
say that the most significant lesson learned regarding : 
is that it works at all. Perhaps one of the greatest ve..-- 
of the ARPANET experiment was to show that a dist.-.Vii 
control adaptive routing algorithm would indeed conw::ji 
routes which were sufficiently good. The difficulty in f - 
this lies in the fact that we are dealing with a dynai:::. ■ ■ 
tion in a distributed control environment with delay r - 
feedback paths for control information flow. The 
proof that things do work has had an important intra¬ 
net work design; indeed, these distributed algorithm 
currently operating successfully in a number of paeVc 
works. 

D: Lessons from Broadcast Channels 

As mentioned earlier, packet communications has fouaJ i 
portant applications in the areas of satellite packet b* i,J -- ■ 
ing and in ground radio packet switching. In both cn-.- 
ments we have a situation in which a common 
channel is available to be shared by a multiplicity - 
Since these users demand access to the channel at imr---' 
able times, we must introduce some access schr--- 
coordinate their use of the channel in a way which r rT 
degradations and mutual interference. In many ot the 
described [10]- we have found that “burst” comm uni ' J -‘ 
provides efficiencies over that of “trickle” transnussior- 
this we mean that when a data source requires access ■- 
channel, it should be given access to the full capacity 
broadcast channel and not be required to transmit at * 
speed using only a fraction of the available band»N-- 

Section V-A on principles regarding “bigger is better )■ 

In examining the recent literature, we find that a nu^ 
access schemes have been invented, analyzed. 


ami puf 
[io; 
of 


for a summary of many of these access schemes, see 
observe that these access schemes fall into one 
categories, each with its own cost. The first of these 1 
random access contention schemes whereby little or no ‘ 
is exerted on the users in accessing the channel, and tW 
in the occasional collision of more than one packet, 1 *■ ^ 
destroys the use of the channel for that transmit 0 ” 
schemes as pure ALOHA, slotted ALOHA, and ( l0 J 
lesser extent) Carrier Sense Multiple Access t au ^ 
category. At the opposite extreme, we have the staIiC ib 
tion access methods which preassign capacity to usert 
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“dedicated” as opposed to multiaccess channels. Here 
'Problem is that a bursty user will often not use his preas- 
*j opacity in which case it is wasted. Such schemes as 
Division Multiple Access and Frequency Division Multiple 
fall in this category. Between these two extremes are the 
■” ; c reservation systems which only assign capacity to a 
hen he has data t0 send. The loss here is due to the 
^H-ad implementing the demand access. Such schemes 
~filing (where one waits to be asked if he has data to send), 
1 , reservation schemes (where one asks for capacity when 
' ., f ds it), and Mini-Slotted Alternating Priority (where a 
1 ., is passed among numbered users in a prearranged se- 
^-.v, giving each permission to transmit as he receives a 
all fall in this category. Each of these schemes pays 
-hute to nature as shown in Table II. 

•fortunately, at this point in time we are unable to evaluate 
„ minimum price (i.e., a degradation to throughput and/or 
_j, i one must pay for a given distributed multiaccess broad¬ 
er rnvironment. 

t: have found that contention schemes are fundamentally 
-;ihle in that they have a tendency to drift into a congested 
riiere the throughput decreases significantly at the same 
_... ;he delay increases. Fortunately, however, we have been 
-x io design and implement amazingly effective control 
which stabilize these contention schemes [20]. An- 
lesson we have learned is that certain tempting ways of 


th a dyniunS^iiJiursf two access schemes (e.g., taking a fraction of the traffic 
with delays Is (Ma ^ 1 faction of the capacity assigned to one access scheme, 
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using that capacity to handle that traffic using a second 
_.-e scheme) does not give an improvement in the overall 
::jghput-delay performance [10]. Furthermore we have 
~J that certain capture effects exist in some of the con- 
- ; on schemes (e.g., a group of terminals may temporarily 
i :he system capacity and thereby “lock out” other groups 
rtcnded periods of time) and one must be wary of such 
-i.jincna [20], 

have also found that in a ground radio broadcast environ- 
r -'. a few buffers in each packet radio unit appear to be 


:ommon bro»l4Kf ,:,,,:lcn t to handle the storage requirements [21]; this comes 


■“‘•i large!;, due to the fact that our transceivers are half- 

. —,-^• e '’ tbe y can either transmit or receive, but not both, 

access scbcflt W* ‘ P vc| i time). We can show (see Section IV-E) that dedicated 
ray which pjt*^w r4ius t channels have an inherent advantage over dedicated 
any of thel®*^ ^networks in a large (many-user) bursty store-and-forward 
;” commuaW** ^-‘-nment [22], Moreover, we have investigated the optimal 
trahsmistWf ® ^Tihsion range for ALOHA networks and have found that 
uires aceedWfcf ^broadcast networks cam be made quite effective when 
ill capacity^®® * -raffic is not bursty; ihdeed-fhis optimal range is chosen so 
transmit *t * ^ ^channel utihzation in the resulting local ALOHA system 
,le bandwid®^ ‘ -‘'and t hen those networks need only \fe more capacity 
is better”)^/ ■ ' c ' lhe corresponding M/M/1 network [22]. 
d that a nuaj* we point out that perhaps one of the first applications 

ed, and " t0J dcast radio access schemes will be to implement these 

mes, see (lyl* ■ ^; themes on wire networks (for example, coaxial cables 
nto one ofyjj™ --^-optics channels) in a local environment; an example of 
;t of these in implementation is the Ethernet [23]. 
little or no , 
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number of. nodes in a network) grows, the cost of 
topological design of such a network behaves like 
~ f ^ is typically in the range from 3 to 6. Thus we 
‘‘■rd! ^Poiogioal design quickly becomes unmanageable, 
we note that as N grows, the size of the routing table 
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m the network grows linearly with N and this too 


TABLE II 

The Cost of Distributed Resources 


Access 

Method 

Collisions 

Cpntrol 

Overhead 

Idle 

Capacity 

Random access contention 

Yes 

No 

No 

Dynamic reservation 

No 

Yes 

No 

Fixed allocation 

No 

No 

Yes 


places an unacceptable burden on the storage requirements 
within an IMP. In addition, the transmission and processing 
costs for updating such large tables is prohibitive. Third, even 
were the design possible, the cost of the lines connecting this 
huge number of nodes together grows very quickly unless 
extreme care is taken in that design. In all three cases just 
mentioned, one finds that the use of hierarchical structures 
saves the day. In the design case, one may decompose the net¬ 
work into clusters of nodes, superclusters of clusters, etc., 
designing each level cluster separately. This significantly reduces 
the number of nodes involved in each subdesign, thereby 
reducing the overall design cost significantly. For example, a 
100-node net would have a cost on the order of 100 4 = 10 8 
(for E = 4), whereas a 2-level hierarchical design with 5 clusters 
would cost on the order of 5(20) 4 + 5(4) < 10 s , yielding an 
improvement of three orders of magnitude! The same approach 
may be used in routing, where names of distant clusters, rather 
than names of distant nodes, are used in each routing table, 
thereby reducing the table length down from A to a number as 
small as e In N giving a significant reduction [ 24 ]. For example, 
a 1000-node net would give a 50-fold reduction in the routing 
table length when hierarchical routing is used. 

In [22] we discuss the overall effect and gain to be had in 
the use of hierarchically designed wire networks and broadcast 
networks. For example, we can show that in a bursty dedicated 
broadcast environment, the use of hierarchical network struc¬ 
tures (even with fixed allocation schemes) yields a system cost 
which is proportional to [log Af] 2 , where M is the number 
of users. Comparing this to the case of wire networks where 
the cost is proportional to the \/m, we see the significant ad¬ 
vantages that broadcast channels have over wire networks in 
a bursty environment when hierarchical structures are allowed. 
We can see this intuitively since we assume that the cost of a 
broadcast channels is proportional only to capacity, but is 
independent of distance; if we properly select the transmission 
range, then the broadcast capacity can be reused spatially (i.e., 
it can be used independently and simultaneously in more than 
one area). Further, it can be shown that a 2-level hierarchy 
using random access in the lower level and dedicated channels 
in the upper level can be quite efficient in a broadcast environ¬ 
ment; this is true since the lower level has gathered together 
enough traffic so that it is no longer bursty when delivered to 
the upper level (recall that dedicated channels do well with 
nonbursty traffic) 

V. Principles Established 
This section is really a continuation of the last since there is 
a somewhat fuzzy boundary between lessons and principles. 
Indeed, one might accept the pragmatic definition that a 
principle is a lesson you had to learn twice. 

A. Bigger is Better 

The law of large numbers states that a large collection of 
demands presents a total demand which is far more predictable 
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than are the individual demands. We are thus led to the con¬ 
sideration of large shared resources (large in the sence that we in¬ 
crease both the number of users—or the load presented by each 
user-and the capacity of the resources). Furthermore it is 
easy to show that the performance improves significantly as 
we make our systems larger. In particular we can show that 
a small system whose capacity is C operations per second and 
whose throughput is J jobs per second (with each job requir¬ 
ing an average of K operations per job) performs A times as 
slowly (i.e., the response time is A times longer) as a system 
whose capacity is AC and whose throughput is AJ. The lesson 
here is very clear, namely that bigger systems perform far better 
than smaller ones [25]. This is a statement about performance 
and not one about cost. Indeed if one is talking about com¬ 
munication channel capacity, then one usually also gains 
through an economy of scale due to the tariff pricing structure 
as presented by the common carriers. All the more reason, 
therefore, to concentrate more and more traffic on ever larger 
channels to gain both cost and efficiency in performance; of 
course one must be careful not to abuse any “resale” restric¬ 
tions. Moreover, our lesson about burst communications tells 
us that in sharing this large channel dynamically, one should 
provide the full capacity to a single user on demand, rather 
than to preallocate fractions of the capacity on a permanent 
basis (omitting consideration of such channel-sharing schemes 
as spread-spectrum). 

The “bigger is better” principle may not apply to the case 
of stream traffic (defined as real-time traffic which requires a 
low delay and moderately large throughput requirement-an 
example being packetized speech). Indeed, an unresolved 
issue recently raised by Dr. Robert E. Kahn (Editor of this 
Special Issue) is how effective it would be to handle stream 
traffic by dividing each trunk into a multiplicity of medium- 
capacity channels which may then be linked together to form 
a stream traffic path. We are currently looking at this issue. 

B. The Switch 

Our second principle has to do with the use of a software 
switch at the nodes of a network. The principle here is that 
it pays to place intelligence at the switching nodes of a net¬ 
work since the cost of that intelligence is decreasing far more 
rapidly than the cost of the communications resource to which 
it is attached. The idea is to invest some cost in an intelligent 
switch so as to save yet greater cost in the expensive com¬ 
munications resource. The ability iAjntroduce new programs, 
new functions, new topologies, new nodes, etc., are all enhanced 
by the programmable features of a clever communications pro¬ 
cessor/multiplexer at the software node. 

C. Constraints 

The principle here is simply, “constraints are necessary and 
often are evil.” Indeed some of the constraints we have seen 
are sequencing, storage management, capacity allocation, speed 
matching, and other flow and routing control functions. These 
“natural” constraints render us vulnerable to dangerous dead¬ 
locks and degradations. As mentioned above, if the constraint 
cannot be met due to some possibly unfortunate accident, 
than the system will stop ail flow. If one is slow in meeting 
the constraints, then that represents a delay-throughput 
degradation. As a result of this principle, we see that it be¬ 
hooves us to provide sufficient resources in the network which 
then allow us to be more relaxed about assigning them. That 
is, the more precious is a given resource, the tighter we are in 


allocating it to a demand, the more likely we are to ru- 
deadlock or degradation. With an ample resource w 
more cavalier in assignment and even renege on the assig.^ 
if necessary, assuming that a backup facility (ir, the f or 
ample resource elsewhere in the network) is provided 

D. Distributed Control 


The principle here is that one must pay a price to nan.., 
organizing a collection of distributed resources into a coo-.,. 
ing group. We have not yet established what that mir '-. 
price is, but we have classified the price in the form „< . 

sions, control overhead, and idle capacity. 


E. Flow Control 


The “principle” here is that flow control is a critical ' • 
tion in packet communications and we are still naive : . 
invention and analysis of flow control procedures, ilop-rtj. 
cleaner code and cleaner concepts will simplify our abiic, 
design and evaluate flow control procedures in the 
There is a “miniprinciple,” which seems to be emerging h,,- 
preliminary studies [ 26 ] which states that if one \w.v, , 
maximize the power in a network at fixed cost, where ;■ ,,, 
is defined as throughput divided by response time, then 
simple statistical assumptions on the flow, one should 07 . 1 : 1 ,. 
at a point where the throughput delivered is half the iunic„> 
possible and the response time is then twice the minimum :. 
load) response time. 


F. Stale Protocols 


In our experiments in the ARPANET, the SATNET. ar.: > 
packet radio network, we have occasionally attempted in j :■ 
a protocol from one network directly over into a new iui * ■> 
We have found that this is a dangerous procedure and - m 
be carefully analyzed and measured before one adopts ' .• 
procedure. Indeed, the use of old protocols in a new or.v_-» 
ment is dangerous. For example, we found that the use e 
ARPANET-like RFNM end-to-end protocol was eMicy- 
wasteful of channel capacity and resulted in a capture r! M 
between pairs of users when used in the SATNET. In a • 
Time Division Multiple Access scheme (in which odd-numb 
slots are permanently assigned to user A and even-num 
slots to user B), user A could prevent B from sending an. 
if he simply started transmitting first in each of his slot* 
this would require B to devote all of his slots to rciu. 
RFNM’s to A. Time in the SATNET is divided into fixed 1 
slots (of 30-ms duration). A slot is used for a single > J ^ 
transmission even if the packet itself is tiny, as is the . 
RFNM. This inefficiency does not exist in the ARP ^ 
since no extra bits are stuffed into ARPANET pa'A* ^ 
artificially increase their size. Indeed, gateways have _ 
troduced between the ARPANET and SATNET which 
these nets independent of each other’s protocols and >■ 

[ 2 ], 

G. Inexperienced Designers 


It is important that users recognize the difference m ^ ^ 
tion, performance, and operation of a packet neiw ^ 
opposed to a leased line. Certain decisions re gar ^ 
parameter settings in any process-to-process c° niniun ^_ 
are often left up to the user of a packet network; f° r 
the buffer allocation he provides in his HOST to 
from another process communicating through the ^ ^ 
with his HOST is a decision often left to the system uS * r _*ir 
buffer allocation is too small, he may degrade the 3 
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^finance of the network to an unacceptably low degree; 
.QjncS about, not because the network is slow, but rather 
uS e his allocation was too small. The principle here is that 
, 3i leaves design decisions in the hands of the users (or even 
(iork designers) then those individuals must be informed as 
effect of their decisions regarding these parameter set- 
p .. [hey cannot be expected to understand the consequence 
. ; ir actions without being so informed. 

VI. Conclusions 


purpose of this paper has been to boil down a decade of 
whence with packet communications and from this to ex- 
... some lessons and principles we have established. We have 
' ; eded only in part in this endeavor; the field is still moving 
> a critical fu* and we are 1earn * n S new things each day. Indeed, in 
till naive foV_;ion to lessons and principles, we have identified a number 
J res. HopefyR) 
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jen issues which require further study. Aside from the 
;er principles we stated in the preceding section, we feel it 
pessary to make some concluding statements. First we 
, that one must view packet communications as a system 
.•;rthan as a trivial leased line substitute. The use of packet 
-niunications offers opportunities to the informed user on 
... .ine hand and sets traps for the naive user on the other. It 
•.pessary that the overriding principles which we have 
If the irurhftfift Lolished and others which we have yet to establish be well 
e rninimurnfim L::stood by the practitioners in the field. We must continue 
m ram from our experience, and alas, that experience is often 
L-.-J through mistakes observed rather than through clever 
►•action. In all of our design procedures we must constantly 
»are of the opportunity to share large resources among 
|t-rr populations of competing demands. We must further be 
rared to incorporate new technologies and new applications 
|> :iiey arise; we cannot depend upon “principles” as these 
nples become invalid in the face of changing technologies 
k applications. 

mly, we must point out that the true sharing of processing 
-ties in the network (i.e., the HOSTS) has not yet been 
'--;ed in modern day networks. One would dearly love to 
■mit a task to a network, ask that it be accomplished in the 
lui efficient fashion, and expect the network to find the 
'i suitable resources on which to perform that task. Cur¬ 
dy, one must specify on which HOST his program should be 
where his job should be executed, where to store his 
at which location his results should be printed, and 
■’■'fy when all this must happen. The next phase of network- 
niust address this general question, .of automatic resource 
among HOSTS in a distributed processing environment. 

■-Jps in the next special lss>je_Ofl .packet communications we 
' ^ in a position to identify dessons and principles for true 
*' " rce sharing of this type. 


\TNET, ts4 0» 
impted lottap 
a new netwat 
edure and «* 
,e adopts tud t 
a a new ante* 
at the use of fl* 
was extra*# 
a capture c£*f 
4ET. InaJ«* 
h odd-nuabcMf 

. cven-nunb#d 

lending any 4# 
of his slots** 
ots to aluilBf, 
into fixed k** 

r a 

.sis the etf* 
the AW0 
\IET pad38 * 
ys havebw** 1 
;T which re** 
:ols 

■ferenceifl^l 
;ket net*#**§ 
is regard!#* 


References 

Kleinrock, Queueing Systems . Volume II: Computer Applica¬ 
tions. New York: Wiley Interscience, 1976. 

• Jacobs, R. Binder, and E. V. Hoversten, “General purpose 


packet satellite networks,” this issue, pp. 1448-1467. 

(3] R. E. Kahn, S. A. Gronemeyer, J. Burchfiel, and R, C. Kunzelman, 
“Advances in packet radio technology,” this issue, pp. 1468-1496. 

(4] P. Baran, “On distributed communications,” RAND Series 
Reports, Rand Corporation, Santa Monica, CA, Aug. 1964. 

(5] D. Davies, “The principles of a data communication network for 
computers and remote peripherals,” in Proc. IFIP Congress '<55, 
Edinburgh, Scotland, pp. 709-714; Aug. 1968. 

(6] L. G. Roberts, “Multiple computer network development to 
achieve resource sharing,” in Proc. ACM Symp. Operating Syst., 
Gatiinburg, TN, 1967. 

(7] L. G. Roberts, “The evolution of packet switching,” this issue, pp. 
1307-1313. 

(81 L. Kleinrock, Communication Nets; Stochastic Message Flow and 
Delay. New York: McGraw-Hill, 1964, out of print. Reprinted 
by Dover Publications, 1972. (Published in Russian, 1970, 
Published in Japanese, 1975.) 

[9] O. J. Boxma, “On a tandem queueing model with identical service 
times at both counters, I.,” University Utrecht, Dept, of Mathe¬ 
matics, Preprint No. 78, Mar. 1978. 

(10) L. Kleinrock, “Performance of distributed multi-access computer- 
communication systems,” in Proceedings of IFIP Congress ’77 
Toronto, Canada, pp. 547-552; Aug. 1977. 

(11] L. Pouzin and H. Zimmerman, “A tutorial on protocols,” this 
issue, pp. 1346-1370. 

(121 L. Kleinrock, “ARPANET lessons,” in Proc. Int. Conf. Com¬ 
munications, Philadelphia, PA, pp. 20-1-20.6; June 1976. 

[13] R. Kahn and W. Crowther, “Flow control in a resource sharing 
computer network,” in Proc. 2nd IEEE Symp. Problems in 
Optimization of Data Communication Systems, Palo Alto, CA, pp. 
108-116, Oct. 1971, (also reprinted in IEEE Trans. Communica¬ 
tions , pp. 539-546; June 1972). 

[14] E. Raubold and J. Haenie, “A method of deadlock-free resource 
allocation and flow control in packet networks,” in Proc. Third 
Int. Conf. Computer Communication, Toronto, Canada, pp. 483- 
487; Aug. 1976. 

[15] W. Naylor, “A loop-free adaptive routing algorithm for packet 
switched networks,” in Proc. Fourth Data Communications Sym., 
Quebec City, Canada, pp. 7.9-7.14; Oct. 1975. 

[16] A. Segail, P. M. Merlin, and R. G. Galiager, “Arecoverable protocol 
for loop-free distributed routing,” in Pro. Int. Conf. Communica¬ 
tions, Toronto, Canada, vol. 1, pp. 3.5.1-3.5.5; June 1978. 

[17] H. Opderbeck and L. Kleinrock, “The influence of control proce¬ 
dures on the performance of packet-switched networks,” in Nat. 
Telecommunications Conf. Record, San Diego, CA., pp, 810-817; 
Dec. 1974. 

[18] W. Price, “Simulation of packet*switching networks controlled on 
isarithmic principles,” in Proc. Third Data Communication Symp., 
St. Petersburg, EL, pp. 44-49; Nov. 1973. 

[19] W. Naylor and L. Kleinrock, “On the effect of periodic routing 
updates in packet-switched networks,” in Nat. Telecommunica¬ 
tions Conf. Record , Dallas, TX. dd. 16.2-1-16.2-7; Nov. 1976. 

[20] L. Kleinrock and M. Gerla, “On the measured performance of 
packet satellite access schemes,” in Proc. Fourth Int. Conf. Com¬ 
puter Communication, Kyoto, Japan, Sept. 1978. 

[21 ] F. Tobagi, “Performance analysis of packet radio communication 
systems,” in Nat. Telecommunications Conf. Record, pp. 12.6-2- 
12.6-7; Dec. 1977. . 

(22 ] G. Akavia, “Hierarchical organization of distributed packet-switch¬ 
ing communication systems,” Ph.D. Dissertation, Computer Sci¬ 
ence Department, Univ. of California, Los Angeles, Mar. 1978. 

[23] R. Metcalfe and D. Boggs, “Ethernet: Distributed packet switch¬ 
ing for local computer networks,” Communications-of the ACM, 
vol. 19, no. 7; pp. 395-404, July 1976. 

[24] L. Kleinrock and-F. Kamoun, “Data communications through 
large packet-switching networks,” in Proc. Int. Teletraffic Con¬ 
gress, Sydney, Australia, pp. 521-1-521-10; Nov. 1976. 

[25] L. Kleinrock, “Resource allocation in computer systems and 
computer communication networks,” in Proc. of IFIP Congress 
‘74, Stockholm, Sweden, pp. 11-18; Aug. 1974. 

[26] -, “On flow control,” in Proc. Int. Conf. Communications, 

Toronto, Canada pp. 27.2-1 to 27.2-5, June 1978. 

[27] V. G. T. Cerf and P. Kirstein, “Issues in packet network intercon¬ 
nection,” this issue, pp. 1386-1408. 


commj 

ork;fot 


ta& 




































































